Skip to content

feat: add advisory catalog sentinel - #101

Merged
0xzr merged 1 commit into
mainfrom
codex/resolve-open-issues-20260729
Jul 29, 2026
Merged

feat: add advisory catalog sentinel#101
0xzr merged 1 commit into
mainfrom
codex/resolve-open-issues-20260729

Conversation

@0xzr

@0xzr 0xzr commented Jul 29, 2026 •
edited by sourcery-ai Bot
Loading

Copy link
Copy Markdown
Owner

Closes #67.

What changed

  • add a weekly/manual catalog sentinel that discovers public provider model listings without using user override URLs
  • persist bounded lifecycle state for catalog presence/absence, probe health, threshold crossings, and recovery
  • add opt-in, strictly token-bounded completion canaries with secret-map authentication and no secret/response serialization
  • publish bounded JSON/Markdown artifacts and open or update trusted advisory issues only on actionable drift
  • document operator behavior, baseline semantics, environment protection, and local reproduction
  • let maintenance probes explicitly preserve an 8-token cap without changing the default reasoning-model floor

Acceptance mapping

  • weekly/manual schedule, concurrency, timeouts, least permissions, artifacts, and cached state: .github/workflows/catalog-sentinel.yml
  • public discovery, safe normalization, lifecycle tracking, canaries, reports: scripts/catalog_sentinel.py
  • operator contract and alert semantics: docs/CATALOG_SENTINEL.md
  • fail-first coverage of discovery drift, timeouts, trust boundaries, thresholds, recovery, and probe privacy: tests/test_catalog_sentinel.py, tests/test_client.py

Verification

  • 913 tests passed
  • combined coverage 83.73% (project line gate >=80%; branch coverage remains above the repository >=70% gate)
  • focused sentinel/client tests: 49 passed
  • Ruff: clean
  • mypy (strict, local source): clean
  • Bandit high/high: no findings
  • zizmor high/high: no findings
  • catalog validation: 24 providers, 222 enabled chat routes, 407 cataloged chat models
  • build, twine check, and fresh-wheel smoke: passed
  • security exception policy: 0 active; suppression scan: none
  • live Pollinations discovery normalized 5 IDs and detected the expected stale catalog entry
  • adversarial slow-header request was terminated by the total deadline
  • trusted issue updater requires exact title, Actions bot author, and immutable body marker

Review gate

Codex Sol 5.6 xhigh approved exact git tree 696d2a0ebd0ba1d1d9df2dace1e79bc40cb92f24 with no P0/P1/P2 findings before commit. A separate exact-PR-head xhigh review is required after all GitHub checks pass and before merge.

Summary by Sourcery

Introduce an advisory catalog sentinel workflow and bounded completion probes for detecting catalog drift without mutating routing or provider configuration.

New Features:

  • Add a catalog sentinel script that performs bounded public model discovery and authenticated completion probes with lifecycle tracking and drift reporting.
  • Add a scheduled and manually dispatchable GitHub Actions workflow to run catalog discovery and protected probes, persist state, and open/update trusted advisory issues.
  • Document the catalog sentinel operator contract, environment protection model, and local reproduction steps for discovery and probes.

Enhancements:

  • Allow callers to opt out of enforcing the reasoning-model max_tokens floor so strictly quota-bounded maintenance probes can retain an 8-token cap.

CI:

  • Add a catalog-sentinel GitHub Actions workflow with least-privilege permissions, bounded timeouts, caching of prior state, and artifact publication for public discovery and protected probes.

Tests:

  • Add comprehensive tests for catalog sentinel discovery, probe behavior, lifecycle state handling, issue body generation, workflow configuration, and operator documentation.
  • Add a client test ensuring the thinking-model floor can be disabled for strictly bounded canaries.

@sourcery-ai

sourcery-ai Bot commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Reviewer's Guide

Implements a weekly/manual advisory catalog sentinel workflow and script that safely discovers public provider model listings, runs strictly bounded authenticated completion probes, persists lifecycle state for drift detection, and publishes sanitized JSON/Markdown artifacts and GitHub issues without mutating routing, plus a small client change to optionally disable the reasoning-model max-token floor and corresponding tests and docs.

Sequence diagram for authenticated probe using bounded client.call

sequenceDiagram
  actor GitHubActions
  participant catalog_sentinel as catalog_sentinel.probe
  participant freellmpool_client as freellmpool.client.call
  participant ProviderAPI

  GitHubActions->>catalog_sentinel: main argv=["probe", ...]
  catalog_sentinel->>catalog_sentinel: load_secret_map
  catalog_sentinel->>catalog_sentinel: load_catalog
  loop for each selected provider/model
    catalog_sentinel->>freellmpool_client: call(provider, model.name, _PING, max_tokens=8, enforce_thinking_floor=False)
    freellmpool_client->>ProviderAPI: HTTP completion request
    ProviderAPI-->>freellmpool_client: HTTP response
    freellmpool_client-->>catalog_sentinel: Reply
    catalog_sentinel->>catalog_sentinel: probe_record
  end
  catalog_sentinel->>GitHubActions: write_outputs report, summary
  GitHubActions->>catalog_sentinel: issue-body --report probe.json
  catalog_sentinel-->>GitHubActions: write_issue_body probe-issue.md
Loading

File-Level Changes

Change Details Files
Allow callers to bypass the reasoning-model max-token floor for strictly bounded maintenance probes.
  • Add an enforce_thinking_floor parameter with a default of True to the client.call function.
  • Gate the thinking-model token floor logic on enforce_thinking_floor so bounded probes can keep an 8-token cap.
  • Add a regression test covering disablement of the thinking floor for canary calls.
 src/freellmpool/client.py
tests/test_client.py
Introduce a catalog sentinel script that performs bounded public model discovery and authenticated completion probes, tracks lifecycle state, and generates advisory reports and issue bodies.
  • Add catalog_sentinel.py with public discovery that fetches provider /models endpoints, normalizes model IDs strictly, classifies HTTP outcomes, and maintains absence streak and drift metadata.
  • Implement authenticated probe logic that selects configured providers via a secret JSON map, runs 8-token ping completions with enforce_thinking_floor disabled, classifies failures, and tracks repeated failures and recovery without exposing secrets or responses.
  • Provide helpers for bounded JSON fetching with tight transport timeouts, safe loading and validation of previous state and secret maps, drift detection flags, and rendering of JSON/Markdown artifacts and GitHub issue bodies with advisory-only semantics and strict size caps.
  • Expose a CLI with discover, probe, and issue-body subcommands wiring catalog loading, bounds validation, environment usage, and artifact writing.
 scripts/catalog_sentinel.py
Add comprehensive tests for the catalog sentinel’s classification, lifecycle, privacy, workflow, and documentation behavior.
  • Test HTTP classification, model listing normalization (including Pollinations aliases), and that transient/incomplete listings never recommend retirement.
  • Verify discovery lifecycle merging with prior state, baseline initialization semantics, catalog gap handling, absence streak tracking, threshold crossings, and recovery.
  • Cover probe lifecycle including classification of empty completions, repeated failure/recovery drift behavior, sanitization of exceptions and reports, and strict secret-map validation.
  • Assert that the GitHub workflow and operator documentation contain required advisory, permissions, environment protection, and behavior guarantees.
 tests/test_catalog_sentinel.py
Add a scheduled, least-privilege GitHub Actions workflow to run the catalog sentinel and surface drift as advisory issues.
  • Create catalog-sentinel.yml with weekly schedule and manual dispatch, separate jobs for public discovery and authenticated probes, and conservative timeouts.
  • Configure jobs to install the runtime, restore bounded lifecycle state from cache, run catalog_sentinel.py discover/probe with previous state, and upload sanitized JSON/Markdown artifacts.
  • Implement issue-body generation via the script and use gh CLI to open or update a single drift/probe issue keyed by exact title, Actions bot author, and immutable body marker.
  • Scope permissions to contents read and issues write, use a protected environment for authenticated probes, and ensure workflow is fork-safe by avoiding PR triggers and state mutation.
 .github/workflows/catalog-sentinel.yml
Document the catalog sentinel operator contract, environment protection, and advisory semantics, and link it from contributor docs.
  • Add docs/CATALOG_SENTINEL.md describing public discovery bounds, protected probe configuration via FREELLMPOOL_SENTINEL_KEYS_JSON, lifecycle and artifact behavior, and local reproduction commands.
  • Update CONTRIBUTING.md to reference the catalog sentinel documentation and to clarify that sentinel output is advisory and never authorizes automatic catalog mutation.
  • Note the new advisory catalog sentinel feature in CHANGELOG.md under Added.
 docs/CATALOG_SENTINEL.md
CONTRIBUTING.md
CHANGELOG.md

Assessment against linked issues

Issue Objective Addressed Explanation
#67 Add a scheduled, advisory catalog sentinel workflow that runs weekly and via workflow_dispatch, performing public discovery with bounded timeouts/body sizes, environment-protected authenticated probes, least-privilege permissions, bounded concurrency, and fork-safe secret usage. ✅
#67 Implement catalog sentinel tooling that discovers public model-list endpoints and runs authenticated completion canaries to produce sanitized JSON and concise Markdown artifacts with lifecycle metadata (e.g., last_discovered, last_verified, verification_count, free_tier_kind, billing_risk, region/privacy notes, failure classification), handles incomplete/partial listings and provider-wide errors, and opens advisory issues on drift without auto-enabling/disabling routes. ✅
#67 Add tests and documentation for the catalog sentinel covering incomplete listings, provider-wide errors, sanitization and secret handling, lifecycle/no-auto-mutation behavior, failure reporting (429, 402, quota, transient), and operator/maintainer contract. ✅

Possibly linked issues

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@0xzr 0xzr mentioned this pull request Jul 29, 2026
Closed
7 tasks
sourcery-ai[bot]
sourcery-ai Bot reviewed Jul 29, 2026
View reviewed changes

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The scripts/catalog_sentinel.py module is quite large and mixes CLI plumbing, HTTP interaction, lifecycle logic, and rendering; consider splitting it into smaller, focused modules under src/freellmpool/ and importing them from a thin script entrypoint for better reuse and maintainability.
  • The new enforce_thinking_floor flag in client.call is only lightly documented; it may be worth clarifying its semantics in the docstring and type hints (e.g., when callers should disable it and any risks) to avoid accidental misuse in future probe-like call sites.
  • The sentinel workflow and script both encode numeric bounds (timeouts, max bytes, provider/model limits); consider centralizing these limits in code or configuration so changes don’t require updating multiple places and to keep the operational behavior easier to reason about.
Prompt for AI Agents 
Please address the comments from this code review:

## Overall Comments
- The `scripts/catalog_sentinel.py` module is quite large and mixes CLI plumbing, HTTP interaction, lifecycle logic, and rendering; consider splitting it into smaller, focused modules under `src/freellmpool/` and importing them from a thin script entrypoint for better reuse and maintainability.
- The new `enforce_thinking_floor` flag in `client.call` is only lightly documented; it may be worth clarifying its semantics in the docstring and type hints (e.g., when callers should disable it and any risks) to avoid accidental misuse in future probe-like call sites.
- The sentinel workflow and script both encode numeric bounds (timeouts, max bytes, provider/model limits); consider centralizing these limits in code or configuration so changes don’t require updating multiple places and to keep the operational behavior easier to reason about.
Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@0xzr
0xzr merged commit a3ac8b5 into main Jul 29, 2026
14 checks passed
@0xzr
0xzr deleted the codex/resolve-open-issues-20260729 branch July 29, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sourcery-ai sourcery-ai[bot] sourcery-ai[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add a scheduled catalog sentinel and model lifecycle metadata

1 participant

@0xzr